Inflection 77: Adding C and C++ API for stating source grammemes and tests for the same.#174
Inflection 77: Adding C and C++ API for stating source grammemes and tests for the same.#174grhoten merged 15 commits intounicode-org:mainfrom
Conversation
| * @param value - The speakable string to convert to a concept | ||
| * @param intitialConstraints - The intitial constraints for the map. | ||
| */ | ||
| InflectableStringConcept(const SemanticFeatureModel* model, const SpeakableString& value, const std::map<SemanticFeature, ::std::u16string> intitialConstraints); |
There was a problem hiding this comment.
The map should be a const & for the API.
| * @param speakFeature The speakFeature from the SemanticFeatureModel that represents the SemanticFeature for the speak information for a SpeakableString. | ||
| * @param constraintMap The intitial constraint map. | ||
| */ | ||
| DisplayValue(const SpeakableString& value, const SemanticFeature& speakFeature, const ::std::map<SemanticFeature, ::std::u16string> constraintMap); |
There was a problem hiding this comment.
The map should be a const & for the API.
| DisplayValue::DisplayValue(const SpeakableString& value, const SemanticFeature& speakFeature) | ||
| : DisplayValue(value.getPrint(), {}) | ||
| DisplayValue::DisplayValue( | ||
| const SpeakableString& value, | ||
| const SemanticFeature& speakFeature, | ||
| const ::std::map<SemanticFeature, ::std::u16string> constraintMap) | ||
| : DisplayValue(value.getPrint(), constraintMap) | ||
| { | ||
| if (!value.speakEqualsPrint()) { | ||
| constraintMap.emplace(speakFeature, value.getSpeak()); | ||
| this->constraintMap.emplace(speakFeature, value.getSpeak()); | ||
| } | ||
| } | ||
|
|
||
| DisplayValue::DisplayValue( | ||
| const SpeakableString& value, | ||
| const SemanticFeature& speakFeature) | ||
| : DisplayValue(value.getPrint(), {}) | ||
| { | ||
| if (!value.speakEqualsPrint()) { | ||
| this->constraintMap.emplace(speakFeature, value.getSpeak()); | ||
| } | ||
| } |
There was a problem hiding this comment.
Can you please declare these functions in the same order as the header? It makes it easier to navigate.
Also the 2 argument constructor with the speakFeature can now call the new constructor with an empty map. Then the redundant if statement can be removed. It's best to keep the copy and paste code to a minimum to improve the maintenance.
| InflectableStringConcept::InflectableStringConcept(const SemanticFeatureModel* model, const SpeakableString& value) | ||
| InflectableStringConcept::InflectableStringConcept( | ||
| const SemanticFeatureModel* model, | ||
| const SpeakableString& value, | ||
| const ::std::map<SemanticFeature, ::std::u16string> intitialConstraints | ||
| ) | ||
| : super(model) | ||
| , value(value) | ||
| , defaultDisplayValue(value, *npc(super::getSpeakFeature()), intitialConstraints) | ||
| { | ||
| } | ||
|
|
||
| InflectableStringConcept::InflectableStringConcept( | ||
| const SemanticFeatureModel* model, | ||
| const SpeakableString& value | ||
| ) |
There was a problem hiding this comment.
Can you please declare these functions in the same order as the header? It makes it easier to navigate.
Also the 2 argument constructor with the SpeakableString can now call the new constructor with an empty map. Then the copied code can be removed. It's best to keep the copy and paste code to a minimum to improve the maintenance.
There was a problem hiding this comment.
Sure I would keep that in mind.
Being a fresher I wasn't aware.
|
@grhoten I have added the C API too can you please review and tell me the next steps. |
| * This is set to a failure when a failure has occurred during execution. | ||
| */ | ||
| INFLECTION_CAPI IDInflectableStringConcept* iinf_createWithConstraints(const IDSemanticFeatureModel* model, const IDSpeakableString* value, | ||
| const inflection::dialog::SemanticFeature* features, const char16_t** values, int32_t SemanticFeaturesLen, UErrorCode* status); |
There was a problem hiding this comment.
C++ API is not allowed in C API. Please see ipron_createWithDefaults for how to pass in such values. That API also ensures that the 2 lengths are exactly the same.
For consistency with the PronounConcept API, it would be ideal if this was also named iinf_createWithDefaults to be consistent with the naming.
grhoten
left a comment
There was a problem hiding this comment.
All new API need tests. The code should always be in a releasable state. Please add tests for these changes. The current code should be well over 90% line code coverage.
Please also state which issue is being fixed with these changes. That is best done with prefixing your commits and pull request summary with "Inflection-77".
| #include <inflection/dialog/SemanticFeatureConcept.h> | ||
| #include <inflection/dialog/SemanticFeatureModel.h> | ||
|
|
||
| #include <inflection/dialog/SemanticUtils.hpp> |
There was a problem hiding this comment.
This internal C++ header should not be used in public C API.
| if (pronounData->numValues() == 0) { | ||
| throw ::inflection::exception::IllegalArgumentException(u"Display data can not be empty."); | ||
| } | ||
|
|
There was a problem hiding this comment.
I don't think that this file needs to change.
There was a problem hiding this comment.
Yup sure, I mistakenly added a line and forgot
|
@grhoten
|
|
@grhoten
|
static functions are not available outside of the file. It also compiles. So that implies that the changes are OK.
As long as there are a couple of tests, that should be sufficient to show the utility of the API.
I recommend including a test of iinf_createWithDefaults in InflectableStringConceptTest-c.cpp.
|
| <test><source definiteness="definite">cometa</source><initial gender="masculine"/><result number="singular" gender="masculine">el cometa</result></test> | ||
| <test><source definiteness="definite">cometa</source><initial gender="feminine"/><result number="singular" gender="feminine">la cometa</result></test> | ||
| <test><source definiteness="definite">QQQQ</source><initial gender="masculine" number="plural"/><result number="plural" gender="masculine">los QQQQ</result></test> | ||
| <test><source definiteness="definite">QQQQ</source><initial gender="feminine" number="plural"/><result number="plural" gender="masculine">las QQQQ</result></test> |
There was a problem hiding this comment.
Let me correct my comment. The structure looks generally good, but the last test is wrong. The initial gender is feminine. So the gender result should be feminine too. This looks like a copy and paste error.
| } | ||
| return new SpeakableString(result); | ||
| } | ||
|
|
There was a problem hiding this comment.
@grhoten
Was this the expected behavior in this function? If not, could you please point out where I am going wrong?
There was a problem hiding this comment.
No. Within the DictionaryLookupFunction, you're only returning the grammeme value. This does not need to change.
There was a problem hiding this comment.
DictionaryLookupFunction::getFeatureValue likely needs to be updated to query the relevant values from displayValue. It should probably query DisplayValue::getFeatureValue or DisplayValue::getConstraintMap
Okay sure I wanted to know exactly what I need to do, based on your previous comment I came up with this commit, could you please tell what is expected to make the tests pass so that i could implement the same and further add tests in InfectableStringConceptTest-c.cpp as well to complete this task ?
There was a problem hiding this comment.
Can you please revert the change to this file. It looks like it's only changing the empty lines.
…createWithDefaults as InflectableStringConceptTest-c#testCreateWithDefaults
|
Hello!! |
| auto initialConstraint = initialConstraintMap.find(feature); | ||
| if (initialConstraint != initialConstraintMap.end()) { | ||
| return new SpeakableString(initialConstraint->second); | ||
| } |
There was a problem hiding this comment.
These changes are confusing. Why are these changes needed?
There was a problem hiding this comment.
More precisely, why is getFeatureValue being called here? Why is the feature name being checked for gender? What's special about gender or number? Why not grammatical case?
There was a problem hiding this comment.
auto displayConstraint = displayConstraintMap.find(feature);
if (displayConstraint != displayConstraintMap.end()) {
auto numberFeature = npc(getModel())->getFeature(u"number");
if (feature.getName() == u"gender" && baseValue && numberFeature != nullptr
&& displayConstraintMap.find(*npc(numberFeature)) != displayConstraintMap.end()) {
return baseValue.release();
}
return new SpeakableString(displayConstraint->second);
} This is the test case which was failing before the changes:
<test>
<source definiteness="definite">cometa</source>
<initial gender="masculine"/>
<result gender="masculine" number="singular">el cometa</result>
</test>-
InflectableStringConcept::getFeatureValueasks the default display function for a renderedDisplayValue, so feature lookups reflect whatever the display layer injected (articles, inflections, etc.). Spanish determiners add bothgenderandnumberto that map, and the gender can contradict the lexeme default (la cometa). -
To avoid reporting the article’s guess as the noun’s gender, the code detects
feature == "gender"and, when the number constraint is present too, returns the lexeme’s base value instead of the display-layer value. English articles never set gender, and no language mutates case in this path yet (I am not sure though), so only the gender/number pair needs a guard today. -
InflectableStringConcept::getFeatureValueasks the default display function for a renderedDisplayValue, so feature lookups reflect whatever the display layer injected (articles, inflections, etc.). -
Spanish determiners add both
genderandnumberto that map, and the gender can contradict the lexeme default (la cometa).
To avoid reporting the article’s guess as the noun’s gender, the code detectsfeature == "gender"and, when the number constraint is present too, returns the lexeme’s base value instead of the display-layer value, as these conditions when satisfied point to the case when we have an article being inserted. -
Grammatical case isn’t treated specially because the Spanish article logic never synthesizes case information—so there’s no regression to guard against. If a language later injected case-altering determiners we’d need a similar safeguard, but today only the gender/number pair exhibits the mismatch.
There was a problem hiding this comment.
This behavior does not scale, and it does not make sense in my mind. An InflectableStringConcept is just a simplified SemanticConcept. A SemanticConcept should not have this issue. It does not have this kind of logic, and it does not check on specific grammatical category values.
There was a problem hiding this comment.
Yes, you're right it is just a general constructor and logic, I think the underlying locale specific synthesizer needs to be addressed for the failing test case.
|
@grhoten instead to get the include path automatically. |
| add_library(xml2 INTERFACE IMPORTED GLOBAL) | ||
| set_target_properties(xml2 PROPERTIES IMPORTED_LIBNAME xml2) | ||
| target_include_directories(xml2 INTERFACE ${CMAKE_OSX_SYSROOT}/usr/include/libxml2) | ||
| find_package(LibXml2 REQUIRED) |
There was a problem hiding this comment.
This seems like a good improvement.
| #include <inflection/npc.hpp> | ||
|
|
||
| std::map<std::u16string, std::u16string> XMLUtils::getAttributes(xmlNodePtr node) { | ||
| if(node == nullptr) return { }; |
There was a problem hiding this comment.
Please split this line and use curly braces. It makes it easier to place break points.
|
The only test failing is this one: <test><source definiteness="definite">QQQQ</source><initial gender="feminine" number="plural"/><result number="plural" gender="masculine">las QQQQ</result></test>This is a spanish-specific issue in which the article has association with gender and the Could you suggest some next-steps? |
|
@grhoten Is the discussion going in the wrong direction or if you could suggest some other way? We only need to address this issue for the failing test |
| auto inflectableConcept = iinf_createWithDefaults(model, | ||
| sourceString, | ||
| defaultConstraints, | ||
| static_cast<int32_t>(sizeof(defaultConstraints) / sizeof(defaultConstraints[0])), |
There was a problem hiding this comment.
Please change this calculation to use std::ssize instead. It's a C++20 feature.
|
|
||
| #include <inflection/npc.hpp> | ||
| #include <inflection/dialog/InflectableStringConcept.hpp> | ||
| #include <inflection/dialog/SemanticFeatureConcept.h> | ||
| #include <inflection/exception/ClassCastException.hpp> | ||
| #include <inflection/util/ULocale.hpp> | ||
| #include <inflection/dialog/SemanticFeature.hpp> | ||
| #include <inflection/dialog/SemanticUtils.hpp> | ||
| #include <inflection/dialog/SemanticFeatureModel.hpp> | ||
| #include <inflection/util/TypeConversionUtils.hpp> | ||
| #include <inflection/util/Validate.hpp> |
There was a problem hiding this comment.
Generally, I've tried to keep the first line the header for the current file, followed by an empty line, sort the headers, and leave npc.h last.
Can you please sort the headers in this section?
| constraintMap.emplace(*npc(getSpeakFeature()), value.getSpeak()); | ||
| } | ||
| SemanticFeatureModel_DisplayData displayData({DisplayValue(value.getPrint(), constraintMap)}); | ||
| SemanticFeatureModel_DisplayData displayData({defaultDisplayValue}); |
There was a problem hiding this comment.
For future reference to self and other reviewers, the DisplayValue constructor with a SpeakableString takes care of the speak usage. It didn't disappear.
There was a problem hiding this comment.
@grhoten
Yes, in the latest commit I did the following change in the InflectableStringConcept::getDisplayValue:
As their are initial constraints as well we might need to pass the constraintMap which was constructed into the defaultDisplayValue. I am not sure, please validate this step, Although the tests are passing.
There was a problem hiding this comment.
If this step is correct i will resolve the macos test issue and remove the lines used for debugging so that the PR is ready to merge.
Sorry for the delay. The changes are in the right direction now. Once the new comments are addressed, then these changes should be ready to merge. |
| * Copyright 2021-2024 Apple Inc. All rights reserved. | ||
| */ | ||
| #include "catch2/catch_test_macros.hpp" | ||
| #include <iostream> |
There was a problem hiding this comment.
Please remove the logging changes with iostream. Please use Catch for logging any relevant information. The tests should run clean by default so that it's easier to find relevant information for failing tests.
There was a problem hiding this comment.
Yes actually I just wanted to be sure that the changes I did to InflectableStringConcept::getDisplayValue are correct I mean using the constraintMap from defaultDisplayValue so that we use the initial constraints as well, please validate the overall changes to that function from all commits . And then I will do a final cleanup commit.
There was a problem hiding this comment.
Everything else looks fine, except for the comments that I provided about an hour ago.
| const auto displayValueResult = getDisplayValue(true); | ||
| if (displayValueResult) { | ||
| return npc(defaultFeatureFunction)->getFeatureValue(*displayValueResult, constraints); | ||
| return npc(defaultFeatureFunction)->getFeatureValue(*displayValueResult, constraints); |
There was a problem hiding this comment.
No need for whitespace after the semicolon.
| // auto constraintMap = defaultDisplayValue.getConstraintMap(); | ||
| // if(constraintMap.find(feature) != constraintMap.end()) { | ||
| // return new SpeakableString(constraintMap[feature]); | ||
| // } |
There was a problem hiding this comment.
Please don't leave commented out code in your changes. It makes it harder to read.
| <test><source definiteness="definite">cometa</source><initial gender="feminine"/><result number="singular" gender="feminine">la cometa</result></test> | ||
| <test><source definiteness="definite">QQQQ</source><initial gender="masculine" number="plural"/><result number="plural" gender="masculine">los QQQQ</result></test> | ||
| <test><source definiteness="definite">QQQQ</source><initial gender="feminine" number="plural"/><result number="plural" gender="masculine">las QQQQ</result></test> | ||
| <test><source definiteness="definite">QQQQ</source><initial gender="feminine" number="plural"/><result number="plural" gender="feminine">las QQQQ</result></test> |
|
I am not sure about the macos test issue, please look into the same and other than that the code is ready to merge. |
grhoten
left a comment
There was a problem hiding this comment.
The macOS issue is a separate issue. ICU 78 has different behavior, and it requires the tests to be updated. The curly quote has to be switched to an ASCII straight quote.
We will have to update that in a separate pull request.
|
@grhoten I think this PR is also ready to be merged. I am really interested in the inflection project and look forward to contribute more in the future. |

@grhoten
Issue #77
Could you please review this so that I could move forward for C API and XML parsing too.